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A  Computer  Terminal  Network  for  Transparent  Stimulation 
of  the  User  of  an  On-Line Retrieval  System 

Siegfried  Treu 

A    computer  terminal  network  to  enable  "transparent  stimulation"  of 
the  user  of  an  on-line  retrieval  system  has  been  designed,    implemented, 
and  pilot  tested.     Its  basic  purpose  is  to  provide  a  suitable  and  effective 
framework  and  methodology  for  experimental  identification/validation 
of  those  human  characteristics  which  should  be  recognized/reinforced 
in  man-computer  interface  design.      The  rationale  behind  the  transparent 
stimulation  approach  is  presented  and  the  methodology  employed  for  such 
real-time,    unobtrusive  scanning  and  manipulation  of  the  man-computer 
dialogue  is  described.     A  general  overview  of  the  hardware  and  soft- 
ware features  of  the  implemented  stimulation  network  is  included. 

Key  words:     Computer  terminal  network;  human  characteristics; 
interactive  information  retrieval;  man-computer  interface;  transparent 
stimulation;  unobtrusive  monitoring. 


IV 


1.     INTRODUCTION 

While  man's  communication  with  other  men  often  defies 
rational  explanation,   the  study  of  man- computer  communication 
should  be  comparatively  easy.     In  the  latter  case,    an  interested  third 
party  (or  observer)  can  at  least  impose  a  degree  of  control  on  the  non- 
human  half  of  the  communicating  pair. 

Nevertheless,    more  than  a  decade  has  passed  since  Licklider's 
widely  quoted  paper  1  on  "Man- Computer  Symbiosis,  "  and  we  must 
certainly  acknowledge  that  the  goal  of  adequately  understanding  such 
a  complementary  relationship  remains  elusive.      That  is  not  to  say 
that  no  progress  has  been  made,   but  rather  that  much  more  research 
and  experimentation  is  essential. 

A  sample  selection  of  references  to  the  open  literature  can 
briefly  characterize  what  has  been  accomplished  so  far.      The  need  for 
human  factors  research  in  conjunction  with  computer  systems  has  been 
variously  recognized  and  discussed  (e.g.    Mayer^,    Parsons-^). 
Different  modes  of  interacting  with  the  computer  have  been  compared 
experimentally  (e.g.  Sackman^),    as  have  different  types  of  inter- 
active terminals   (e.g.    Carlisle^).      Probably  more  attention  has  been 
focussed  on  a  computer's   response  time  to  various  user  requests  than 
on  any  other  measure  of  interactive  computer  performance  (e.g. 
Carbonell  et  al    ,    Miller    ,    Cordaro  and   ChienS).  While  response  time 
seems  to  have  a  definite  bearing  on  user  satisfaction,    certain  other 
characteristics  are  also  indicated  by  collective  results  of  experi- 
mental studies  conducted  thus  far  (as   reviewed  by  Sackman"). 

Some  researchers  have  attempted  theoretical  characterization 
or  modelling  of  man-computer  problem- solving  (e.g.    Horman^). 
Others  have  taken  a  deliberate  laboratory  approach  of  providing  a 
research  staff  with  innovative,    interactive  hardware  and  software 
tools  intended  to  facilitate  communication  (e.g.    Engelbart  and 
English^).  So,    the  man-computer  interface  has  not  been  neglected 
as  such.     Our  increasing  awareness  of  it  is  obviously  highlighted  by 
the  fact  that  a  complete  chapter  (Bennett    ^)  will  be  dedicated  to  it  in 
the   1972  Annual  Review  of  Information  Science  and  Technology. 

But  many  of  the  human  characteristics  which  should  be 
reinforced  at  the  interface  are  either  not  yet  identified  or  not 
experimentally  verified.      This  is  partly  due  to  the  lack  of  suitable 


and  effective  methodology.      With  this  in  mind,    the  author*  designed, 
implemented  and  tested  a  computer  terminal  network  enabling 
"transparent  stimulation"  of  a  user  during  his  interaction  with  a 
computer.      This   Technical  Note  describes  the  resulting  seven-month 
project. 

Emphasis  is  on  presenting  the  rationale  behind  the  stimulation 
approach  and  on  portraying  the  various  requirements  for  such  a  net- 
work.    Only  a  general  overview  of  the  hardware  and  software  character- 
istics is  provided.      The  more  voluminous  details  are  described  else- 
where in  an  internal  NBS  report.     An  outline  account  of  the  transparent 

stimulation  approach  is  also  available  from  a  1971  AFIPS  Workshop 
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paper1 D . 

2.     RATIONALE  BEHIND  THE  STIMULATION  APPROACH 

2.  1     Need  for  Interface  Improvement 

It  is  a  commonly  acknowledged  fact  that  computer  systems  in 
general,    and  interactive  computer  systems  in  particular,   have  usually 
been  designed  by  computer  specialists  who  were  far  more  interested  in 
taking  advantage  of  the  technical  capabilities  of  the  system  than  in 
meeting  the  requirements  of  the  human  user.      Thus  the  user-computer 
interface  has  been  relatively  neglected,    at  least  from  the  user's  point- 
of-view.     We  may  have  considerably  better  typewriter-like  terminals 
available  now,    and  more  and  more  users  are  gaining  access  to  various 
types  of  graphics  terminals.     We  may  also  have  much  more  effective 
interactive  software  available  to  the  on-line  user.      But  that  still  leaves 
us  far  short  of  the  interface  improvements  which  may  be  possible  if  we 
could  only  learn  to  understand  and  verify  the  natural  human  character- 
istics which  should  be  complemented  or  reinforced  at  the  interface 
with  a  computer,    and  which  should  consequently  be  incorporated  in 
further  iterations  of  interface  design. 

If  both  the  need  and  potential  for  interface  improvement  are 
indicated,   how  is  it  possible  to  arrive  at  a  definition  of  those  elusive 
human  characteristics  which  must  be  taken  into  account?     Should  we 
ask  the  users  for  their  opinions? 


-While  on  an  NRC-NBS  Postdoctoral  Research  Associateship  in  the 
NBS  Center  for  Computer  Sciences  and  Technology  (CCST)  with  leave- 
of-absence  from  the  Department  of  Computer  Science,    University  of 
Pittsburgh. 


2.  2     Questionable  Dependence  on  Users 

Supervisors  of  computer  users  normally  are  (or  should  be) 
interested  in  how  productive  the  users  are  (or  could  be)  when  inter- 
acting with  computer  systems  for  various  purposes.      From  that 
standpoint,    it  may  be  that  a  user  should  not  necessarily  be  provided 
instantaneous  response  time,   for  example,    in  order  to  be  most 
effective.      Perhaps,    as  considered  by  Boehm  et  al^4}    a  "lockout"  or 
deliberate  delay  in  response  can  enhance  the  problem- solving  thought 
processes   required  for  certain  computer-aided  tasks.      The  user  may 
be  frustrated  by  such  delays,   but  he  might  be  more  productive. 

On  the  other  hand,    since  we  live  in  a  society  in  which  the  rights 
and  well-being  of  the  individual  should  be  important,    we  should  reckon 
with  that  individual's  preferences  in  the  context  of  computer  assistance. 
This  does  not  have  to  be  detrimental  to  productivity,    since  we  know  that 
an  interdependence,    albeit  very  complex  in  nature,   between  productivity 
and  human  satisfaction  exists.     If  we  want  the  computer  user  to  be  both 
effective  and  satisfied,   we  should  consider  him  personally  to  be  a 
valuable  source  of  data  for  experimental  determination  of  his  levels  of 
effectiveness  and  satisfaction. 

v 

But  how  should  this  source  of  information  be  tapped?     Conventional 
questionnaire  and  interview  techniques  have  frequently  been  used  in  the 
past.      These  have,   however,    been  variously  criticized  (especially  in  the 
area  of  information  storage  and  retrieval)  as  amounting  to  opinion  polls 
of  questionable  dependability.     Humans  are,    after  all,    recognized  as 
having  numerous  inconsistencies,    memory  failures ,   biases,    etc.,    which 
tend  to  have  strong  bearings  on  how  they  might  fill  out  a  questionnaire. 

At  another  extreme,    a  user's  subjective  responses  could  be 
altogether  evaded  by  making  physiological  measurements  on  him  during 
his  interaction  with  a  computer.     In  this  manner,    the  user's  fluctuations 
in  emotional  response   (e.  g.    indicated  by  rate  of  heart  beat),    as  precipi- 
tated by  "computer  system  behavior,  "  could  be  correlated  or  compared 
with  known   symptoms  of  frustration,   happiness,    etc.     It  is  not  clear, 
however,    that  the  average  user  (especially  when  serving  as  an  experi- 
mental subject)  becomes  emotionally  affected  enough  during  his  inter- 
action with  a  computer  to  provide  physiological  indicator  values  which 
could  reliably  discriminate  among  different  levels  of  satisfaction. 
Nevertheless,    this  technique     might  become  useful  at  some  time  in  the 
future,    especially  when  applied  to  computer  users  during  periods  of 


real-life  crises.     At  least  it  could  serve  to  verify  subjective  verbal 
responses,    or  vice  versa.      For  purposes  of  this  transparent  stimu- 
lation project,   however,    its  application  was  neither  considered 
feasible  nor  desirable. 

One  more  major  approach  to  gaining  meaningful  data  about 
user-computer  interaction  is  to  monitor  and/or  observe  that  inter- 
action.    As  discussed  in  the  next  section,    the  necessary  choice  is 
between  doing  so  either  actively  or  passively. 

2.  3    Active  versus  Passive  Monitoring 

Passive  monitoring  of  a  user-system  dialogue  affords  the 
collection  of  data  about  what  the  user  in  fact  experiences   (in  terms  of 
response  time,    system  functions,    etc.  )  as  he  makes  various  demands 
within  the  limitations  of  the  particular  system's  capabilities.     However, 
it  does  not  lend  itself  to  gaining  the  corresponding  levels  of  user 
satisfaction  and  requirement,    except  by  implication  (e.  g.    if  he  uses  a 
certain  feature  often  enough,   he  seems  to  like  or  need  it). 


Part  of  the  problem  could  be  resolved  by  passive  monitoring 
complemented  by  an  on-line  facility  for  questioning  a  user  about  his 
current  level  of  satisfaction,    immediately  after  a  significant  system 
response  has  occurred.      In  this  manner,    any  wants  or  needs  implied 
by  the  collected  data  on  actual  user  demands  could  at  least  be  verified 
(or  refuted)  by  a  user's  personal  judgment,    taken  immediately  after 
the  event  of  concern  and  thus  at  a  comparatively  advantageous  point  in 
time   (within  the  user's  span  of  short-term  memory). 

But  this  semi-active  monitoring  approach  still  has  its  problems 
A  particular  system  is  by  design  limited  to  a  certain  type  of  response 
under  a  certain  set  of  circumstances.      The  number  and  nature  of 
alternatives  in  a  system  response  at  any  time  is,    therefore,    quite 
restricted.     If  we  happen  to  be  interested  in  studying  and  redesigning 
the  entire  man- computer  interface,   based  on  the  spectrum  of  pertinent 
human  characteristics,   we  must  have  greater  flexibility  and  versatility 
in  what  takes  place  at  the  interface. 

This  is  where  active  monitoring  and  observation,   herein  called 
"transparent  stimulation,  "  comes  into  play.      To  test  a  user's  reactions 
to  various  alternative  system  responses,   we  must  be  able  to  stimulate 
him  by  causing  alternate,    preplanned  system  responses  to  occur  at  the 
appropriate  times.      This  stimulation  can  be  made  contingent  on  a 


combination  of  software  scanning  and  human  (observer)  monitoring  of 
the  dialogue.     It  must  of  course  be  unobtrusive  or  "transparent"  to 
the  user,    if  at  all  possible.     And  it  must  be  coupled  by  the  already 
described  mechanism  for  requesting  and  receiving  on-line  indications 
of  user  satisfaction  or  emotion. 

Should  the  monitoring  and  stimulation  capability  be  implemented 
by  modifying  the  object  computer  system  itself,    or  by  utilizing  a 
separate,    intermediary  processor?     Because  of  the  prominent  arguments 
against  manipulating  an  operational  system  for  research  purposes,    it 
is  currently  preferable  to  employ  a  totally  separate  research-oriented 
computer,    if  such  is  available.     Such  a  computer  system  can  then  be 
operated  as  an  "invisible"  intermediary  between  user  and  object  system, 
with  no  perturbing  effects  on  the  latter. 

One  might  furthermore  ask  why  the  available  research-oriented 
computer  is  not  also  employed  as  the  object  system  itself,    thus 
precluding  any  need  for  accessing  an  operational  system.      This 
alternative  is  generally  not  feasible  because  the   research  computer 
may  be  incompatible  with  or  inadequately  equipped  to  satisfactorily 
perform  all  of  the  functions  of  a  real  object  system  and  because  of  the 
added  complexities  of  having  to  simulate  real-life  operational  services. 
As  these  answers  indicate,    the  costly  local  development  of  representa- 
tive retrieval  software  for  research  purposes  appears  to  be  an  even 
less  desirable  approach. 

2.  4    Selected  Computer-Assisted  Task 

Any  intention  to  manipulate  or  modify  a  user- system  interaction 
surfaces  the  requirement  of  having  to  analyze,    in  great  detail,    the 
peculiar  features  and  functions  of  the  system  (and  its  software  sub- 
system^)) being  used.      This  in  turn  brings  out  the  well-known 
differences  in  system  design  based  on  differences  in  computer-assisted 
tasks.     On-line  computer  programming  (sub)systems ,    for  example,    are 
quite  unlike  on-line  information  retrieval  (sub)systems ,   which  are 
unlike  computer-aided  design  (sub)systems . 

Although  the  transparent  stimulation  approach  should  be 
applicable  to  other  computer-assisted  tasks,    the  anticipated  com- 
plexities and  time  constraints  clearly  may  demand  selection  of  and 
focus  on  just  one.      For  this  project  the  choice  was  on-line  information 
retrieval.     It  was  made  because  of  the  author's  particular  interest  in 
that  field  and  also  because  of  comparatively  easy  availability  of  one  or 
more  such  systems. 


2.  5    Establishment  of  Design  Criteria 

What  information  about  the  user  should  the  innovative  designer  of 
an  on-line  retrieval  system  have  in  order  to  develop  and  apply  criteria 
for  improved  interface  design?     At  a  minimum,   he  should  be  informed 
of  the  human  characteristics  of  which  he  should  take  advantage  or 
which  he  should  reinforce.      Particular  characteristics  maybe  applicable 
to  on-line  information  retrieval,   but  they  may  also  apply  more  generally. 

In  the  already  referenced  paper  about  the  conceptual  framework 
for  the  searcher  system  interface  139    a  set  of  twelve  such  character- 
istics were  described  and  summarized  as  follows: 

Designers  of  interactive  search  systems,  in  purposely  planning 
for  an  effective  searcher-system  interface,  must  deliberately  take  into 
account  that  the  human  searcher  has  a  sense  of  (or  need  for): 

1.  Spatial  reference   (or  perspective) 

2.  Order  (or  file  arrangement) 

3.  Completeness   (or  comprehensiveness) 

4.  Association  (or  connectedness) 

5.  Simplicity  (or  clarity) 

6.  Accessibility  (or  convenient  access) 

7.  Responsiveness   (or  prompt  reaction) 

8.  Control  (or  manageability) 

9.  Versatility  (or  variety  in  means  and  modes  of 

access ) 

10.  Compatibility  (or  harmony  among  means  and 

modes  of  access) 

11.  Reliability  (or  confidence) 

12.  Support  (or  advice  and  assistance -on- 

demand) 


These  asserted  characteristics   (and  others  that  could  possibly 
be  added)  can  be  looked  upon  as  subject  to  experimental  confirmation 
or  rebuttal.      They  (or  a  subset  of  them)  can  represent  the  core  of  an 
experimental  hypothesis.     However,    they  must  first  be  embodied  in  a 
set  or  sequence  of  actions,    or  stimuli,     which  after  presentation  to  the 
user  will  elicit  confirming  or  refuting  responses. 

If  such  experimental  work  can  convincingly  confirm  or  refute 
one  or  more  of  the  above  assertions,    then  the  interface  designer 
should  be  better  prepared  to  establish  criteria  for  improved  design. 
The  transparent  stimulation  project  is  intended  to  be  one  step,   however 
small,    in  that  direction. 

2.6     Characteristic  Stimulus  -  Response  Patterns 

Based  on  the  reasoning  behind  the  transparent  stimulation 
approach  presented  thus  far,    three  important  actions  are  seen  to  be 
subsumed: 

1.  the  user-system  dialog  is  monitored  and 
observed 

2.  the  user  is  variously  stimulated 

3.  the  user  is  prompted  to  request  or  react  with 
not  only 

a.  the  normal  system-oriented  message, 
but  also,  whenever  specifically  asked, 
with 

b.  an  indication  of  current  level  of 
satisfaction  as  well  as 

c.  reason  for  that  satisfaction  level 

Since  the  fundamental  question  of  interest  is  "What  does  the 
user  do,    and  how  does  he  feel,    as  a  consequence  of  the  presence  or 
absence  of  a  certain  characteristic  in  a  system  action?",   we  are 
faced  with  trying  to  handle  stimulus -response   (S-R)  pairs  in  the 
context  of  man- computer  interaction.      Consistent  with  the  abundant 
literature  in  behavioral  psychology  concerned  with  stimulus - 
response  behavior  (e.g.    Kimble^),   we  consider  our  experimental 
subject,    namely  the  human  user,    to  be  the  object  of  the  stimulation. 


The  computer  hardware /software  is  then  responsible  for  presenting 
the  appropriate  stimuli  to  the  user,    and  the  user  in  turn  generates  the 
responses.     Although  this  viewpoint  is  contrary  to  the  computer 
scientist's  usual  way  of  considering  the  response  as  coming  from  the 
computer,    it  is  purposeful  in  that  our  priority  focus  is  on  studying  user 
(not  computer)  behavior. 

This  approach  lends  itself  to  the  following  characterization: 

Let  S       =  a  stimulus  generated  by  the  on-line  retrieval 
system  itself 

Sa     =  an  artificial  or  alternate  stimulus  generated 

by  the  intermediary  stimulation  system  or  the 
human  observer 

R      =  an  original  user  request  or  a  user  response 
to  a  stimulus   (S  or  Sa) 

Re    =  a  user  indication  of  stimulus  -  related 
satisfaction  or  emotion 


Rr    =  a  user  indication  of  reason  for  that  satis 
faction  or  emotion 


then  the  data  patterns  of  interest  are  shown  in  Table   1 


(jT) "(R) *~0D **      unjudged    system  stimulus 


s) >h(r) *(Sa) *-      unjudged  artificial  stimulus 


\$)  vfy  \§xT     **      user-judged  system  stimulus 


© — <5) ^ 


user-judged  artificial  stimulus 
^e>-© 


Table   1.      Characteristic  Stimulus -Response  Patterns 
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To  prove  or  disprove  an  experimental  hypothesis  about  user 
behavior,    these  data  patterns  must  be  properly  planned,    collected  and 
then  analyzed.     Such  patterns  can  conceivably  be  embodied  in  more 
complicated  S-R  patterns  to  be  detected  or  observed  in  the  communi- 
cation channel  between  user  and  system,   with  an  accompanying  increase 
in  the  complexity  of  rendering  them  logically  (or  psychologically) 
meaningful  and  useful. 

3.      OVERVIEW  OF  NETWORK  IMPLEMENTATION 

3.  1     General  Framework 

Before  continuing  with  a  presentation  of  how  the  characteristic 
stimulus-response  patterns  are  in  fact  ascertained  and  then  utilized, 
it  seems  appropriate  to  provide  a  general  overview  of  how  the  stimu- 
lation network  was  implemented.     As  stated  in  the  introduction,    most 
of  the  pertinent  details  are  covered  in  a  separate  report. 

The  transparent  stimulation  project,    as  conceived  and 
implemented,   was  strongly  influenced  by  capabilities  which  were  in 
the  process  of  being  developed  by  the  Computer  Systems  Section  of 
the  COST.     In  addition  to  integrating  a  set  of  networking  commands  1° 
into  the  operating  system  software  of  the  locally  available  time-sharing 
system^ ',    thus  enabling  an  applications  program  to  set  up  network- 
like communication  with  a  number  of  different  devices  via  various 
interfaces,    the  Section  was  also  preparing  a  "dialogue  monitoring" 
capability^".      The  latter  allows  a  user  to  dial  into  the  local  (intermediary) 
computer  system  which  then  assists  him  in  establishing  communication 
with  the  remotely  located  time -sharing  system  of  interest  to  him.      The 
intermediary  system  subsequently  monitors  and  records  the  entire  user- 
system  dialog  and,    in  so  doing,    collects  and  analyzes  various  data  (such 
as   response  times)  significant  to  an  evaluation  of  user  demands  and  of 
the  remote  system's  performance. 

The  above-mentioned  networking  capability  is  an  essential 
contributor  to  the  functioning  of  the  applications  software  for  trans- 
parent stimulation.      The  concept  of  dialog  monitoring  is  clearly  related 
to  that  of  monitoring  the  dialogue  and,    whenever  warranted,    stimulating 
the  user  by  means  of  inconspicuous  intervention  or  interjection  of 
variable  messages  and  functions.      The  main  difference  is  that  the 
stimulation  framework  involves  deliberate  manipulation  of  the  interface. 


The  principal  constituents  of  the  stimulation  framework  are  the 
following: 

1.  A  user  -  the  person  wishing  to  use  an  inter- 
active computer  system  for  purposes  of 
accomplishing  one  or  more  information 
retrieval  tasks. 

2.  An  observer  -  the  person  who  is  situated 
and  equipped  to  observe  or  monitor  the  user- 
system  interaction  and  who  initiates  selected 
stimuli  whenever  appropriate. 

3.  An  intermediary  system  -  the  system  or  subsystem 
programmed  not  only  to  assist  the  observer,    by 
means  of  message  interception  and  processing, 
but  also  to  complement  him  by  initiating  certain 
stimuli  on  its  own. 

4.  An  interactive  retrieval  system  -  the  computer 
system  which  is  available  to  assist  the  user  in 
carrying  out  his  information  retrieval  task(s). 

After  the  user  and  the  observer  are  each  provided  with  an 
appropriate  terminal  station,    communication  among  the  four  systems 
or  devices  is  established  as  shown  in  Figure  1.      The  figure     not  only 
portrays  the  communication  flow  but  also  the  various  functions  which 
are  carried  out  within  the  transparent  intermediary  system.      The 
design  includes  two  major  modes  of  transmitting  messages  between 
the  user  and  the  retrieval  system.      The  first  mode  simply  enables  a 
message  to  be  buffered  and  then  forwarded.      This  is  indicated  by  the 
dashed  arrows.      The  second  mode  (solid  arrows)  provides  the  capability 
of  buffering  a  message  (from  user  or  system),    scanning  it  and 
attempting  to  match  it  against  stored  key  word  tables,    and  then  taking 
further  action  depending  on  the  result.     In  case  of  a  miss,    the  message 
is  simply  forwarded.     In  case  of  a  hit,    it  must  first  be  ascertained 
where  in  the  pre-established  S-R-S  hit  pattern  the  interaction  currently 
stands.     If  a  complete  S-R-S  hit  pattern  has  not  yet  been  detected,    the 
stimulation  mode  is  continued  by  forwarding  the  current  message.     If 
such  a  pattern  has  been  detected,    alternate  action  is  called  for. 

This  alternate  action  can  take  three  basically  different  forms. 
By  calling  on  a  selection  of  action  subroutines,    the  intermediary  system 
can: 
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Figure   1.     General  Functions  of  the  Intermediary  System 
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1.  Transmit  a  specific  command  to  the  remote 
system,   which  will  in  turn  initiate  a  new 
stimulus 

2.  Transmit  a  specific  command  or  question 
to  the  user,    thus  in  itself  presenting  a  new 
stimulus 

3.  Request  the  observer  to  input  a  message, 
thus   requiring  him  to  initiate  a  new  stimulus, 
in  either  of  two  forms: 

a.  Direct  command  to  the  remote  system 

b.  Direct  message  or  question  to  the  user 

3.2    Hardware  and  Software  Features 

The  MOBIDIC-B  research  computer  was  employed  as  the 
intermediary  processor  node  in  the  stimulation  network.      The  other 
nodes  are  represented  by  several  teletype  terminals,    two  specially 
constructed  devices,    and  the  remotely  located  information  retrieval 
system  itself.      This  hardware  configuration  is  pictured  in  Figure  2. 

The  research  facility  operates  under  the  locally  developed 
SUPER- CORD  Time-Sharing  System^  which  supports  such  software 
processors  as  an  assembler,    a  BASIC  compiler,    and  the  CL.6  compiler^, 
Although  it  lacks  effective  character  string  manipulation,    the  CL.6 
compiler  was  selected  as  the  most  suitable  for  this  project.     Hence 
the  entire  applications  software  package  was  written  in  CL6,    utilizing 
whenever  appropriate  the  system  networking  commands'"  implemented 
by  Robert  Rosenthal. 

In  order  to  enable  generation  of  previously  described  stimuli, 
contingent  upon  the  occurrence  of  a  certain  user  response   (R)  to  the 
standard  system  stimulus   (S),    a  set  of  seven  action  subroutines  were 
defined  and  implemented  as  part  of  the  CL.6  software  package.      These 
subroutines  have  the  following  functions  within  the  stimulation  network: 

1.      Transmit  Observer  Message  from  Observer  Terminal 
(Device  3  in  Figure  2)  to  User  Terminal  (Device   1) 


Transmit  Observer  Message  from  Observer 
(Device  3)  to  Remote  System  (Device  6). 
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Figure  2.     Hardware  Configuration  of  Stimulation  Network 
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3.  Transmit  Software -Selected  Message  to  User 
Terminal 

4.  Transmit  Software -Selected  Message  to  Remote 
System 

5.  Transmit  User  Message  from  User  Terminal  to 
Observer  Terminal 

6.  Transmit  Software -Selected  Prompt  to  Observer 
Terminal 

7.  Software- Controlled  Set  of  Emotion  Indicator 
Device  (Device  2) 

Because  this  applications  package  requires  the  maximum 
available  user  partition  (20,  480  words  of  36  bits  each),    the  research 
computer  must  be  employed  on  a  dedicated  basis  in  supporting  the 
project. 

3.  3    Special  Indicator  Device 

Two  special  devices,   numbered  2  and  4  in  Figure  2,    were 
designed  and  constructed  for  purposes  of  providing  a  mechanism 
whereby  the  human  observer  (or  the  supporting  software)  can  request 
and  receive  on-line  indications  of  the  user's  level  of  satisfaction  (with 
the  immediately  preceding  system  action)  as  well  as  the  user's 
associated  reason.      The  electronic  componentry  and  terminal  shells 
for  these  devices  were  specified  and  assembled  by  Mr.    William  Truitt 
and  Mr.    James  McNally  of  the  Computer  Systems  Section. 

The  devices  are  purposely  simple  to  operate.      The  user's 
keyboard  has  a  prompt  light  at  its  top  and  two  vertical  columns  of 
seven  labelled  buttons.      Whenever  the  prompt  light  is  lit,    the  user  is 
asked  to  push  one  of  the  left  column  of  buttons  to  indicate  a  level  of 
satisfaction  and  then  one  of  the  right  column  of  buttons  to  give  an 
associated  reason.     The  observer's  special  device  is  quite  similar  in 
surface  format  except  that  he  has  a  prompt  button  where  the  user  has 
a  light,    and  he  has  two  columns  of  labelled  lights  which  are  lit  after 
and  in  correspondence  with  the  buttons  pushed  by  the  user.      The 
observer  can  thereby  gain  immediate  visual  feedback  on  current  user 
level  of  satisfaction. 
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4.     ANALYSIS  OF  REMOTE  SYSTEM  CHARACTERISTICS 

4.  1     Selected  System 

Many  incompatibilities  which  are  well  know  to  exist  among 
different  computer  (hardware  and  software)  systems,    are  also 
prominent  within  the  class  designated  on-line  information  retrieval 
systems.     Most  of  them  have  different  sets  of  commands,    communi- 
cation conventions  and  so  forth.      For  this  and  certain  other  constraints 
(e.g.    local  terminal  capabilities),    it  was  necessary  to  select  and 
utilize  one  particular  system.     After  fairly  extensive  consideration, 
the  choice  was  the  National  Library  of  Medicine's  AIM-TWX  System. 
That  system  offers  a  good  set  of  interactive  commands,    representative 
of  many  retrieval  systems,  and  displays  desirable  consistency  in 
communication  conventions  as  well  as  interaction  language.     It  further- 
more is  accessible  from  standard  teletypewriters  through  the  direct 
dial  network. 

The  AIM-TWX  System  is  fully  described  in  its  user  manual^ 
and,    to  some  extent,    in  a  recent  publication^    . 

4.  2    Message  Format  Modules 

The  interaction  language  for  AIM-TWX  had  to  be  carefully 
analyzed.     In  order  to  carry  out  a  character-by-character  scan  (and 
match)  of  the  messages  transmitted  between  user  and  system,    it  was 
essential  to  not  only  identify  delimiting  characters  but  also  to  find  any 
consistently  recurring  character  string  sequences  or  format  modules. 

This  interaction  language  analysis,    based  on  available  sample 
printouts  and  aided  by  the  previously  mentioned  dynamic  dialog 
monitoring  facility,   led  to  the  format  module  definitions  shown  in 
Table  2.     AIM-TWX  displays  very  helpful  consistency  in  (almost) 
always  preceding  a  system  message  with  the  word  PROG:,    and  by 
(almost)  always  prompting  the  user  with  the  readiness  cue  USER:. 
These  two  format  modules  are  labelled  SI  and  S4  respectively.     If  we 
define  a  "format  module"  to  be  a  detectable  component  (character 
string)  of  a  transmitted  message,    then  a  "module  sequence"  can  be 
regarded  as  a  detectable  succession  (or  string)  of  format  modules 
perhaps  interspersed  with  variable -length,    non- significant  character 
strings . 
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A  frequently  occurring  module  sequence  transmitted  by  AIM- 
TWX  is  S1-S2-S4,    where  the  format  module  S2  represents  the  start  of 
the  systems 's   request  for  a  new  Search  Statement  ("SS").     After  the 
stimulation  software  has  successfully  detected  this  module  sequence, 
it  turns  its  attention  to  scanning  the  user's   response  for  one  of  the 
eligible  response  format  modules. 


The  stimulus  and  response  format  modules  displayed  in  Table  Z 
are  embedded  in  a  number  of  different  possible  module  sequences, 
including  those  displayed  in  Table  3.      The  terminology  is  purposely 
consistent  with  the  S-R-S  patterns  discussed  in  Section  Z.  6.      Thus, 
for  example,    a  system  stimulus  may  be  a  message  beginning  with  SI, 
followed  by  variable  data,    followed  by  SZ,    followed  by  additional 
variable  data,    and  finally  terminating  with  the  prompt  to  the  user, 
namely  S4.      The  user  can  then  respond  with  a  standard  command  Rl 
or  a  search  statement  R3  (unlike  Rl,    not  enclosed  in  double  quotes)  or 
with  an  (operating)  system-enabled  command  RZ.     One  of  these  responses 
in  turn  precipitates  a  system  reaction  in1  the  form  of  another  stimulus. 
It  is  at  this  point  that  the  transparent  stimulation  framework  can  either 
allow  a  legitimate  system  stimulus  or  replace  it  by  an  artificial  one. 
In  either  case,    the  user  may  or  may  not  be  asked  to  indicate  his 
resulting  level  of  satisfaction. 


SEO. 
No. 

S-R-S     SEQUENCE 

1 
2 

3 

4 
5 

— 

SI-S2-S4 

R(lor3) 

- 

SI  ..  . 

- 

SI-S2-S4 

- 

R2 

- 

S5 

— 

SI-S6-S2-S4                  | 

- 

R2 

- 

S5 

— 

SI-S6-S2-S4 

- 

Rllor3) 

- 

SI  ... 

— 

SI-S3-S4 

- 

R4 

- 

SI... 

— 

Table  3.     Possible  Format  Module  Sequences 

17 


4.  3    Screening  and  Matching  Hierarchy 

The  number  of  possible  module  sequences   (of  those  in  Table  3) 
for  which  the  intermediary  system  can  actually  scan  naturally 
depends  on  the  desirable  sophistication  and  available  capacity  of  the 
software  and  hardware  facilities.      For  purposes  of  this  project  it  was 
decided  to  restrict  the  implemented  software  to  sequences  numbered 
1  and  2.     However,    all  five  were  considered  in  initially  prescribing  a 
strategy  for  scanning  user-system  interchanges.     This  strategy  is 
portrayed  in  the  screening  and  matching  hierarchy  of  Figure  3. 

At  the  outset,    the  SI   module  has  to  be  sought  in  the  character 
stream  transmitted  by  the  AIM-TWX  system.     As  soon  as  a  "hit"  is 
registered,   focus  turns  to  finding  S2.     At  this  point,    two  major  options 
can  be  made  available.     In  case  of  S-2  miss,    module  sequences  numbered 
3,    4,    and  5  (in  Table  3)  can  be  pursued;  in  case  of  S-2  hit,    the  next  step 
in  satisfying  module  sequences  numbered  1  and  2  is  taken.     As  already 
indicated,    the  latter  option  alone  was  found  to  be  adequately  complex  to 
implement  for  purposes  of  this  project.     Software  expandability  to 
incorporate  the  other  option,    when  and  if  feasible,    is  however  provided. 

The  elements  of  Figure  3  are  organized  into  clusters,    labelled 
A  through  E,    each  of  which  displays  a  homogeneity  in  terms  of  module 
nature  and/or  length  making  it  suitable  for  creation  of  a  corresponding 
data  file.      These  files  are  of  course  prerequisite  to  the  process  of 
scanning  and  matching  the  user- system  message  stream  against  the 
various  predefined  modules.      Prior  to  the  transparent  stimulation 
procedure,   the  relatively  short  files  are  loaded  into  core,    properly 
structured,    to  enable  dynamic  table  look-up  and  consequent  software- 
controlled  decision- making  on  what  action  to  take  next. 

5.  PRESCRIPTION  OF  TESTABLE   FEATURES 

5.  1     Characterization  of  Commands 

In  order  to  be  able  to  test  any  hypothesis  based  on  the  assertion 
that  certain  human  characteristics   (see  Section  2.  5)  are  significant  to 
the  design  of  a  user-system  interface,    it  is  essential  to  somehow 
characterize  the  available  system  features  accordingly.      Therefore, 
each  of  the  AIM-TWX  commands   (or  command  types)  was  studied  and 
consequently,    using  primarily  subjective  reasoning  or  intuition, 
categorized.      The  categorization  criteria  were: 
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Figure   3.     Hierarchical  Strategy  for  Message  Scanning 
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1.  Primary  or  secondary  applicability  (if  any)  of 

a  particular  human  characteristic  to  the  command 
or  command  type. 

2.  Examinability  (of  the  first  criterion)  at  one  or  more 
of  three  levels  of  complexity. 

Level  of  complexity  refers  to  the  degree  of  involvement  of  the 
intermediary  (stimulation)  system  in  the  remote   (AIM-  TWX)  system 
for  purposes  of  studying  a  particular  characteristic  with  respect  to  a 
specific  command.      The  level  is  labelled  "system-peripheral"  if  a 
characteristic  can  be  studied  independently  of  the  features  of  the 
specific  remote  search  system  and  without  need  for  graphic  manipu- 
lations.     The  level  is   "system-dependent"  when  remote  system  features 
have  to  become  involved,    but  with  no  graphic  interface  manipulation  as 
yet  required.      Thirdly,    the  most  complex  level  is  "file-dependent"  in 
that  both  remote  system  functions  and  features  and  graphic  manipulation 
become  significant. 

This  categorization  of  commands  is  summarized  in  Table  4.      The 
check  mark  (\/ )  indicates  primary  and  the  dot  (  •  )  indicates  secondary 
applicability  of  the  identified  characteristic  to  the  command  group  (left- 
most column)  at  the  corresponding  level  of  complexity. 

5.  2    Selected  Sub -Hypotheses 

Due  to  the  various  time  and  equipment  constraints  on  this  project, 
it  was  decided  to  restrict  the  initial  stimulation  system  design  to  the  first 
two  levels  of  complexity  and,   within  those  levels,    to  the  characteristics 
which  are  underlined  in  Table  4. 

How  is  it  possible  to  hypothesize  anything  about  the  highly  com- 
plex,   serial  nature  of  a  user's  interaction  with  a  computer?     This 
question  is  especially  puzzling  when  there  are  confounding,    cumulative 
effects  to  contend  with,    as  far  as  a  user's  level  of  satisfaction  is  con- 
cerned.    Nevertheless  this  transparent  stimulation  project  is  in  part 
predicated  on  the  belief  that  an  honest  attempt  at  developing  a  different 
approach  and  at  resolving  related  difficulties  is  likely  to  be  more 
beneficial  to  the  man- computer  interface  than  is  not  trying  it  at  all. 

A  sequence  of  "sub -hypotheses "  is  defined  relative  to  the 
categorization  shown  in  Table  4.      These  are  displayed  in  Table  5.      The 
underlying  rationale  for  each  sub-hypothesis  is: 
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IF 

A  CERTAIN  HUMAN 
CHARACTERISTIC  IS 
SIGNIFICANT  TO 
INTERFACE  DESIGN, 
IT  IS  EXPECTED  THAT 


WHEN 
THE  SYSTEM  GIVES 
POSITIVE /NEGATIVE 
REINFORCEMENT  TO 
THAT  CHARACTERIS- 
TIC (AFTER  A  USER 
RESPONSE) 


THEN 
THE  USER 
SHOULD  BE 
STIMULATED 
TO  REACT 
POSITIVELY/ 
NEGATIVELY 
(OR  FAVORABLY/ 
UNFAVORABLY) 


It  was  previously  stated  that  this  project's  focus  is  on  studying 
user  (not  computer)  behavior  in  user-system  interaction,    although  the 
ultimate  objective  is  of  course  to  determine  criteria  for  better  inter- 
face design.     Hence,    in  reference  to  Table  5,    the  above  hypothesis 
format  would  assert,   for  example,    that  if  the  feeling  of  having 
"control"  over  the  system  is  important  to  the  user,   he  would  be 
expected  to  react  negatively  whenever  he  asks  to  have  five  citations 
(known  to  be  available)  to  be  printed  out  and  only  two  are  actually 
printed. 

Most  of  the  AIM-TWX  commands  listed  in  Table  5  are 
abbreviated  for  convenience.      The  system  scans  for  either  full  or 
abbreviated  forms.      The  nature  of  each  artificial  stimulus  generated 
by  the  intermediary  system  (at  times  with  observer  help)  is  necessarily 
only  briefly  indicated.      Whether  or  not  a  user's   reaction  to  such  a 
stimulus  is  positive  or  negative  (or  indifferent)  has  to  be  ascertained 
by  means  of  the  specially  designed  on-line  indicator  device  described 
in  Section  3.  3.     Many  of  the  sub -hypotheses  portrayed  in  Table  5  may 
appear  to  be  either  trivial  or  self-evident.     However,    it  is  not  clear 
that  this  is  necessarily  true  or  that  it  should  preclude  their  experimental 
confirmation  within  the  framework  outlined. 

6.     EXPERIMENTAL  PROCEDURE 

6.  1     User  and  Observer  Environment 

A  special  user  terminal  room,    isolated  from  unwanted  outside 
interference,    provides  the  user  with  privacy  and  also  makes  him  feel 
reasonably  comfortable.     As  previously  indicated,    the  terminal  equip- 
ment available  to  the  user  is  a  Teletype,    Model  33  (because  an 
appropriate  CRT  was  initially  not  available)  and  also  the  specially 
constructed  "satisfaction  level"  device. 
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In  addition,    an  intercom  device  is  installed  to  enable  the  user 
to  request  immediate  assistance,   whenever  essential,    from  the 
observer  who  is  monitoring  the  user  room  by  means  of  an  intercom 
headset. 

A  compact  clustering  of  AIM-TWX  instructions  and  their 
brief  descriptions  is  posted  above  the  teletype  in  easy  view  of  the  user. 
The  experimental  user,   who  is  selected  based  on  having  had  previous 
experience  with  on-line  retrieval  systems,    is  expected  to  be  able  to 
review  the  posted  instruction  repertoire  and  without  much  difficulty 
become  familiar  enough  to  use  it. 

This  task  is  further  simplified  by  posting,    on  one  side  of  the 
user-system  interface,    a  detailed  work  assignment.     It  includes  not 
only  the  exact  sequence  of  steps  to  be  carried  out,    but  also  the 
identity  of  each  instruction  to  be  employed.     The  added  benefit  of 
requiring  such  an  interactive  work  sequence  explicitly  is  the  control  it 
imposes  on  an  experimental  session  for  purposes  of  later  comparison 
of  experimental  subjects. 

To  complete  the  list  of  items  made  available  to  the  user,    a 
copy  of  the  MeSH  Manual  (Medical  Subject  Headings)  is  in  his  vicinity 
for  reference  or  perusal  whenever  necessary. 

The  intermediary  observer    is     located  in  a  separate  area,    the 
computer  room  itself,    and     is    faced  with  the  very  interesting  and 
challenging  task  of  monitoring /manipulating  the  user- system  inter- 
action by  means  of  controlling  all  of  the  following  devices  in  parallel: 

1.  Teletype  Model  35   (Device  3  in  Figure  2) 

2.  O-set  (Device  4) 

3.  Teletype  Model  33  (Device  5) 

4.  Communications  Controller,  for  manual  rerouting 
of  messages  or  message  components  via  different 
interfaces  whenever  the  situation  warrants 

5.  The  Computer  System,  control  console  itself,  for 
recovery  from  various  error  conditions  whenever 
necessary 

6.  Intercom  headset,   for  monitoring  the  user  room 
and  giving  advice  when  essential. 
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6.2    Procedural  Guidelines 

As  evidenced  by  the  above  list  of  devices  he  must  control,    the 
observer  has  a  variety  of  activities  to  monitor  and  attend  to.      To  aid 
him  in  doing  so  and  in  preparing  and  assisting  and  debriefing  the 
experimental  subjects,   he  is  provided  with  a  written  set  of  guidelines 
to  be  referenced  before,    during  and  after  each  session. 

6.  3     Plans  for  Data  Analysis 

The  collection  of  a  number  of  different  types  of  experimental 
data  was  initially  planned.      These  included  the  following: 

1.  User  and  system  response  and  processing  time  data, 
as  collectable  by  means  of  the  previously  described 
Dialogue  Monitor 

2.  Minimum  response  time  variation  data,    using  a 
special  feature  controllable  by  the  observer 

3.  Observer-prompt  to  user  response  times   (via  special 
indicator  devices)  as  tagged  and  printed  out  on  the 
observer  terminal  (Device  3) 

4.  Satisfaction/reason  indicator  data,  as  collected  on- 
line via  the  special  user  device  following  deliberate 
stimulation 

5.  Post- experimental  questionnaire  data,  using  a  pre- 
pared questionnaire 

6.  Less  precise  but  nevertheless  invaluable  record  of 
experiences  gained  by  the  human  observer  in  the 
performance  of  his   role 

The  first  of  the  above  turned  out  to  be  infeasible  because  of  the 
added  complexities   resulting  from  trying  to  run  the  Dialog  Monitor  in 
conjunction  with  the  Transparent  Stimulation  Network.      The  second 
type  of  data,    involving  variation  of  minimum  response  time,    is 
collectable  using  current  software  but  is  dependent  on  a  basically 
different  experimental  design.      The  length  of  the  project  period 
precluded  the  testing  of  more  than  one  design. 
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The  remaining  four  types  of  data  are  applicable  to  the  experi- 
ment that  was  carried  out.     As  confirmed  by  Figure  4,    the  on-line 
satisfaction/reason  indicator  data  are  at  the  core  of  anticipated  data 
analysis.      The  user  can  either  be  given  a  primarily  positive   (P)  treat- 
ment or  a  primarily  negative  (N)  treatment  (see  Table  5),    depending  on 
which  interaction  mode  he  is  in.     Hence,    assuming  the  possibility  of 
random  assignment  of  experimental  subjects  to  one  of  two  groups,    one 
gets  treatments  in  P  -  N  order  and  the  other  in  reverse.     It  was  planned 
that  the  tallying  of  individual  user  indications  into  individual  and  group 
matrices,    such  as  portrayed  in  Figure  4,    should  enable  certain  kinds 
of 

1.  Inter-group  comparison 

2.  Intra-group  comparison 

3.  Inter-treatment  comparison 

4.  Intra -individual  comparison 

(vs.    data  collected  via  questionnaire) 

and  could  result  in  implications   on  the  significance  of  hypothesized 
human  characteristics. 

Perhaps  the  above  sounds  both  questionable  and  presumptuous. 
However,    recognizing  the  well-known  difficulties  in  experimental 
control  and  design  of  such  a  complex  and  confounding  process  as  man- 
computer  interaction,    it  is  nevertheless  felt  that  some  useful  patterns, 
maybe  only  trivial  or  possibly  unexpected,    could  result. 

7.      CONCLUSIONS  AND  RECOMMENDATIONS 

A  total  of  ten  pilot  test  sessions,    involving  five  different  test 
subjects,   have  been  conducted  using  the  transparent  stimulation  net- 
work.     In  the  process,    a  number  of  both  major  and  minor  improvements 
were  made,    ranging  from  significant  smoothing  of  at  times  intolerably 
sporadic  print-out  (on  the  user  terminal)  to  adding  a  more  noticeable 
light  on  the  special  user  device. 

But  three  important  problem  areas  remain.  These  cannot  be 
alleviated  without  major  changes  in  procedure  or  computer  hardware 
or  software: 
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Figure  4.      Overview  of  Plan  for  Data  Analysis 
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Task  complexity  -   Three  of  the  five  test  users 
found  that  the  user-system  interface   (including 
posted  instructions)  was  initially  "overwhelming.  " 
This  may  be  attributable,    at  least  in  large  part, 
to  the  fact  that  the  test  subjects  who  were 
conveniently  available  did  not  meet  the  stipulated 
prerequisite  of  experience  with  on-line  retrieval 
systems  although  they  have  extensive  experience 
with  on-line  systems  in  general.     If  this  explanation 
is  not  born  out  in  further  experimentation  with 
appropriately  experienced  users,    then  significant 
preparatory  training  as  well  as   restructuring  of 
interface  posters  maybe  necessary. 

System  transparency  -   From  the  user's  point  of 
view  there  are  at  times  abnormal  delays  between 
print-out  of  one  message  segment  (or  line)  and  the 
next.      This  kind  of  delay  cannot  only  become  an 
irritation,    perhaps  overshadowing  other  system 
characteristics  of  concern,    but  it  also  detracts  from 
the  transparency  espoused  for  the  stimulation  net- 
work.     The  basic  reason  for  such  delays  lies  in  the 
fact  that  the  message-store-scan-and-forward  mode 
necessitates   "handshaking"  in  the  dual  processor 
while  accomplishing  the  task  of  message  processing 
and  forwarding  simultaneously  to  further  message 
receipt  (in  case  of  long  messages).     A  thorough 
timing  analysis  of  the  dual  processor  system  soft- 
ware could  undoubtedly  lead  to  minimization,    if  not 
total  elimination,    of  this  problem. 

System  reliability  -    Again,    from  the  user's  point 
of  view,    the  system  appeared  to  be  unreliable  too 
often.      This  was  in  part  explained  by  memory  parity 
and  other  such  problems  perturbing  the  particular 
computer  system  (especially  during  humid  summer 
weather).      But  this  was  also  regrettably  due  to  a 
pure  capacity  problem.      Thus  certain  user-issued 
commands,    resulting  in  excessively  long  messages 
coming  from  the  system,    could  literally  flood  (and 
crash)  the  system.     Within  its  hardware  and  software 
constraints,    the  system  could  simply  not  process 
characters  quickly  enough  to  continually  free  up 
storage  for  newly  incoming  characters.     Although 
a  software  safeguard  against  crashing  the  system 
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is  clearly  possible,    the  complete  problem  will 
undoubtedly  not  be  overcome  without  complete 
rewriting  of  the  applications   software   (perhaps  at 
assembly  language  level)  for  a  different  computer 
configuration. 

This  project  has  provided  valuable  experience  on  how  such  a 
transparent  stimulation  framework  can  be  designed  and  implemented. 
It  seems  clear  that  the  above -indicated  problem  areas  can  be  over- 
come.    Although  the  task- specific  experiment  has  only  been  pilot 
tested  so  far,    the  experimental  procedure  as  outlined  appears  to  be  a 
reasonable  first  approximation.      Perhaps  an  attempt  should  be  made 
to  minimize  the  nonintellectual  aspects   (e.  g.    routine  button  control)  in 
the  role  of  the  unobtrusive  observer.     His  job  was  found  to  be  quite 
challenging. 

The  general  stimulation  framework  as  developed  and  described 
herein  holds  considerable  potential  for  research  into  various  character- 
istics of  the  man- computer  interface.      This  conclusion  precipitates  the 
following  recommendations: 

1.  In  general,    experimentation  within  the  environment 
afforded  by  such  a  stimulation  framework  should 
continue . 

2.  More  specifically, 

a.  The  stimulation  support  software  should  be 
reprogrammed  for  a  more  modern  research 
computer  configuration  with  above -indicated 
problems  eliminated.     Graphic  terminals 
should  replace  teletypewriters.     New  research 
computer  equipment  has  recently  been  acquired 
to  augment  CCST  research  facilities  and  plans 
are  being  formulated  for  further  work  of  this 
type  utilizing  these  facilities. 

b.  A  complete  experiment,    still  oriented  to  the 
task  of  on-line  retrieval,    should  be  carried 
out.      This  should  of  course  be  followed  by 
interpretive  analysis  of  the  experimental  data. 

c.  Depending  on  the  success  of  that  experiment, 
either  further  experiments  similar  in  nature 
may  follow,    or  further  study  to  modularize 
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the  stimulation  framework  for  also  handling 
other  types  of  interactive  applications  (e.g. 
CAD&E,    CAI)  might  be  in  order. 

Lastly,    various  features  of  the  stimulation  frame- 
work,   such  as  on-line  collection  of  user  preference 
data  via  special-purpose  devices,    should  be  regarded 
and  reviewed  as  potentially  applicable  to  the  CCST 
Dialogue  Monitor  project,    in  particular,    and  to  the 
entire  CCST  teleprocessing  performance  measurement 
area,    on  the  whole. 
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